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The  Description  of  Large  Systems 

Kent  Pitman 


Abstract 

In  this  paper,  we  discuss  the  problems  associated  with  the  description  and  manipulation  of  large 
systems  when  their  sources  arc  not  maintained  as  single  files.  We  show  why  and  how  tools  that 
address  these  issues,  such  as  Unix  HAKE  and  Lisp  Machine  DEFSYSTEM,  have  evolved. 

Kxisting  formalisms  suffer  from  the  problem  that  their  syntax  is  not  easily  separable  from  their 
functionality.  In  programming  languages,  standard  “calling  conventions"  exist  to  insulate  the  caller 
of  a  function  from  the  syntactic  details  of  how  that  function  was  defined,  but  until  now  no  such 
conventions  have  existed  to  hide  consumers  of  program  systems  from  the  details  of  how  those 
systems  were  specified. 

Wc  propose  a  low-level  data  abstraction  which  can  support  notations  such  as  those  used  by  MAKE 
and  DEFSYSTEM  without  requiring  that  the  introduction  of  a  new  notation  be  accompanied  by  a 
completely  different  set  of  tools  for  instantiating  or  otherwise  manipulating  the  resulting  system. 

l  isp  is  used  for  presentation,  but  the  issues  arc  not  idiosyncratic  to  Lisp. 
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I.  Introduction 


For  reasons  of  modularity  and  editing  convenience,  the  source  code  for  large  program  systems  is 
rarely  maintained  as  a  single  file.  Instead,  it  is  typically  broken  into  a  number  of  smaller  files 
which  together  make  up  the  system. 

Since  system  tools  such  as  editors,  compilers,  loaders  and  printers  tend  to  be  designed  to  deal  with 
files  rather  than  systems,  some  extra  mechanism  is  generally  required  in  order  to  allow  their  users 
to  deal  with  systems  that  span  multiple  files.  In  the  next  section,  we  trace  the  evolution  of  tools 
for  system  building,  identifying  the  important  issues  that  early  tools  sought  to  address. 

We  then  present  an  overview  of  two  system  specification  languages,  Unix*  MAKE  and  Lisp  Machine 
OEFSYSTEM.  which  illustrate  the  level  of  technology  currently  available  to  programmers  for  dealing 
with  these  issues.  Most  criticisms  which  can  be  made  about  existing  tools  arc  at  the  syntactic  level. 
The  tools  address  the  right  issues,  but  their  syntax  can  be  a  stumbling  block,  inhibiting  the 
expression  of  certain  kinds  of  relations  and  the  ability  to  make  extensions  to  the  tools’  original 
functionality.  It  is  not  possible  to  vary  the  syntax  without  rewriting  most  or  all  of  the  underlying 
support 

We  conclude  by  proposing  an  organizational  strategy  which  decouples  syntax  and  functionality. 
'ITiis  makes  it  possible  for  a  programmer  to  develop  alternate  system  maintenance  tools  without 
having  to  reimplement  every  aspect  of  die  original  tools. 

To  make  the  discussion  more  concrete,  we  will  give  several  examples  of  specification  languages 
that  could  be  built  under  die  proposed  framework.  However,  it  is  important  to  understand  that 
the  purpose  of  this  paper  is  not  to  argue  in  favor  of  any  particular  notation.  Rather,  we  wish 
to  illustrate  that  our  proposed  organizational  strategy  establishes  an  appropriate  framework  for 
developing  alternate  notations  such  as  these.  As  an  appendix,  we  offer  a  sample  implementation  of 
these  specification  languages  in  Lisp  Machine  Lisp  in  order  to  further  clarify  any  issues  left 
vague  by  die  examples  used  in  the  body  of  the  paper. 

II.  Background 

There  arc  a  number  of  standard  maintenance  operations  performed  on  systems.  These  operations 
include  (but  arc  not  limited  to)  the  creation  of  hardcopy  listings,  copying  or  renaming  the  files  which 
make  up  the  source  of  the  system,  loading  uncompilcd  source  files  (eg.,  into  a  lisp  interpreter), 
and  compiling  changed  source  files. 

For  a  system  maintained  as  a  single  monolithic  file,  it  is  reasonably  obvious  how  most  of  these 
maintenance  operations  can  be  performed.  Getting  hardcopy  of  the  system  source  is  as  simple  as 
getting  hardcopy  of  any  other  file;  compiling  die  system  is  as  simple  as  compiling  any  single  file. 

For  a  more  complex  system,  where  die  source  spans  more  than  one  file,  performing  these  operations 
may  be  considerably  more  complicated.  In  that  ease,  to  get  a  listing,  one  must  ask  that  each  of 
the  source  files  be  printed.  To  compile  the  system,  one  must  ask  that  each  of  the  source  files  be 
compiled  and.  in  some  cases,  that  the  compilation  occur  in  a  certain  order.  To  load  the  system 
may  be  similarly  complex  due  to  another  (not  necessarily  identical)  set  of  ordering  constraints. 

A  Need  for  Abstraction 

It  used  to  be  that  breaking  a  system  into  multiple  files  meant  that  one  had  to  remember  all  the 
names  of  the  files  and  manipulate  each  as  a  separate  object.  Programming  environments  had  some 
primitive  understanding  of  files  and  operations  to  be  performed  upon  files,  but  had  no  explicit 
understanding  that  groups  of  files  could  work  together  as  a  single  unit 


1  Unix  is  a  trademark  of  Dell  I  ahs. 
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It  was  common  practice  then  (and  still  is  today)  to  create  batch  files  holding  the  commands 
necessary  to  accomplish  a  particular  manipulation.  For  example,  the  programmer  might  have  kept 
a  tile  containing  code  for  compiling  and  loading  a  particular  system  and  another  file  containing 
code  for  hardcopying  its  sources: 

;;  Batch  sequence  to  compile/load  system. 

(LOAD  (COMPILE-FILE  "MACROS .LISP" ) ) 

(LOAD  (COMPILE-FILE  "UTILITY . LISP" ) ) 

( LOAD  (COMPILE-FILE  "MAIN , LISP" )) _ 

COMPILE-AND-LOAD-MYSYS . LISP 


;;  Batch  sequence  to  hardcopy  system  sources. 

(HARDCOPY-FILE  "MACROS . LISP" ) 

(HARDCOPY-FILE  "UTILITY . LISP" ) 

(HARDCOPY-FILE  "MAIM . LISP" ) . 

HARDCOPY-MYSYS .LISP 

Common  though  it  may  have  been  to  manipulate  systems  indirectly  through  such  batch  files,  it  was 
quite  clumsy.  It  meant  that  any  time  a  change  to  the  system  was  made,  all  the  relevant  batch  files 
had  to  be  updated.  If  not  updated  carefully  and  completely,  the  batch  files  could  easily  become 
inconsistent,  leading  to  confusing  effects.  Also,  because  information  about  the  structure  of  a  system 
was  procedurally  embedded,  die  only  way  a  utility  could  be  written  to  apply  a  new  operation  to  the 
system  was  by  creating  yet  another  separately-maintained  batch  file. 

Not  surprisingly,  programmers  have  moved  away  from  this  batch  method  of  maintaining  their 
systems  and  toward  die  notion  dial  a  system  should  be  defined  abstractly  in  some  central  place 
and  then  manipulated  as  a  unit  rather  dian  as  a  set  of  unrelated  individual  components  (files). 

A  First  Approximation 

Here  is  a  simple  example  of  the  sort  of  code  needed  to  implement  the  kind  of  tool  we  are 
discussing: 

(DEFVAR  *SOURCE-INFO*  (MAKE-PLIST) ) 

(DEFUM  SYSTEM-SOURCES  (SYSTEM)  (GET  *SOURCE-INFO*  SYSTEM)) 

(OEFUN  DEFINE-SYSTEM-SOURCES  (SYSTEM  &REST  SOURCES) 

(PUTPROP  * SOURCE-INFO*.  (COPYLIST  SOURCES)  SYSTEM)) 

(DEFUN  HARDCOPY-SYSTEM  (SYSTEM) 

(MAPC  HARDCOPY-FILE  (SYSTEM-SOURCES  SYSTEM))) 

(DEFUN  COMPILE -ANO-LOAO-SYSTEM  (SYSTEM) 

(MAPC  (LAMBDA  (FILE)  (LOAD  (COMPILE-FILE  FILE))) 

(SYSTEM-SOURCES  SYSTEM))) 

Given  this  level  of  support,  a  system  would  be  "defined”  by  writing  simply: 

(OE FINE -SYSTEM-SOURCES  'MYSYS 

"MACROS. LISP"  "UTILITIES. LISP"  "MAIN . LISP" ) . 

Once  defined,  such  a  system  could  be  manipulated  by  requests  such  as  these: 

(IIAROCOPY-SYSflM  'MYSYS) 

( COMPILE -AMD-LOAD -SYSTEM  ’MYSYS) 


Partial  Ordering  of  Dependencies 

One  problem  with  this  formulation  is  that  the  relationship  between  the  modules  of  a  system  may 
be  quite  complex.  For  example,  some  of  die  files  in  a  system  may  not  depend  on  other  files  and 
the  whole  system  may  not  need  to  be  recompiled  just  because  one  file  has  changed.  To  try  to 
account  for  this,  die  definition  of  COMPILE-AND-LOAD-SYSTEM  might  be  changed  to  read: 

(DEFUN  COMPILE-AND-LOAD-SYSTEM  (SYSTEM) 

(LET  ((COMPILE-FLAG  NIL)) 

(DOLIST  (FILE  (SYSTEM-SOURCES  SYSTEM)) 

(SETQ  COMPILE-FLAG 

(OR  COMPILE-FLAG  (NEEDS-COMPILAT ION?  FILE))) 

(IF  COMPILE-FLAG  (COMPILE-FILE  FILE)) 

(LOAD  (BIN-FILE  FILE))))). 

With  this  revised  definition,  files  in  a  system's  source  file  list  would  have  a  left-to-right  dependency 
relation.  Consider  again  the  system  defined  by: 

(DEFINE-SYSTEM-SOURCES  *MYSYS 

"MACROS. LISP"  "UTILITIES. LISP"  "MAIN . LISP" ) . 

If  MACROS. LISP  is  changed.  UTILITIES. LISP  and  MAIN. LISP  will  have  to  be  recompiled. 
But  if  MAIN  .LISP  is  changed,  it  is  the  only  file  which  will  get  recompiled. 

Simple  Icfl-to-riglu  dependency  is  useful  for  some  applications,  but  may  cause  a  lot  of  unneeded 
work  in  others.  The  reason  is  diat  the  actual  ordering  may  only  be  partial,  but  a  full  ordering  is 
forced  by  diis  notation.’ 

For  example,  it  is  easy  to  imagine  our  system  being  constructed  so  that  if  the  file  MACROS.  LISP 
changed,  both  UTILITIES. LISP  and  MAIN.  LISP  would  need  to  be  recompiled,  but  if  just  one 
of  UTILITIES. LISP  or  MAIN. LISP  changed,  only  that  one  file  would  require  recompilation. 

Because  this  simple  notation  provides  no  way  to  adequately  express  such  complex  relations,  it  must 
be  judged  inadequate  to  handle  the  “general  case.”  Nevertheless,  there  arc  eases  where  it  would 
be  adequate,  and  it  would  be  nice  to  use  it  (or  something  equally  simple)  for  those  eases. 

Orthogonal  Dependency  Types 

Another  problem  with  our  original  formulation  concerns  multiple,  independent  dependency  chains. 
Dependency  information  for  compilation  might  not  be  the  same  as  dependency  information  for 
producing  a  runtime  environment. 

If  the  only  goal  is  to  compile  a  system,  there  might  be  many  files  which  do  not  need  to  be 
loaded  because  they  contain  utilities  used  only  at  runtime.  Alternatively,  if  die  goal  is  to  load  an 
already-compiled  system,  some  files  (for  example,  those  containing  only  macros  used  at  compile 
lime)  might  not  be  necessary.  If  both  compilation  and  loading  are  to  be  interleaved,  a  tlvrd 
ordering  might  arise. 

An  adequate  notation  for  describing  systems  needs  to  offer  a  notation  for  staling  different  kinds 
of  dependency  relations,  and  should  probably  be  extensible  (allowing  the  addition  of  new  kinds  of 
dependencies). 


-In  general,  notations  must  he  chosen  with  extreme  care:  an  ill-chosen  notation  can  have  a  very  adverse 
ellcct  on  specifications  that  use  it.  muling  them  seem  to  imply  things  which  are  in  fact  false.  For  a  more 
complete  discussion  of  such  issues.  see  JMuckinlay  X  l|. 


111.  Existing  Tools 


The  Unix  MAKE  Facility 

The  Unix  MAKE  facility  {Feldman  78]  is  frequently  pointed  to  as  a  model  of  “the  right  way  to 
define  a  system.”  It  is  syntactically  simple  and  provides  a  reasonable  amount  of  power. 

A  makefile  contains  Unix  shell  commands  augmented  with  infonnation  about  which  code 
modules  depend  on  which  others.  When  the  make  command  is  invoked,  it  is  as  if  all  the 
shell  commands  were  executed  except  that  some  shell  commands  may  be  “optimized  out”  if  the 
dependency  information  specifics  that  they  arc  not  necessary  to  preserve  correctness. 

To  make  things  concrete,  here  is  a  sample  of  how  a  system  might  be  specified  in  a  makefile: 

mysys:  a.o  b.o 

cc  a.o  b.o  -1m  -o  pgm 

a. o:  Incl  a.c 

cc  -c  a.c 

b. o:  Incl  b.c 

cc  -c  b.c 

The  first  line  defines  that  there  is  some  module  called  mysys  which  needs  to  be  updated  if  a.o 
or  b .  o  arc  ever  out  of  date.  'Ihc  indented  line  following  that  line  specifics  how  to  do  die  update; 
specifically,  it  links  the  compiled  files  a.o,  b.o,  and  standard  library  m.  The  third  and  fifth  lines 
define  modules  a .  o  and  b .  o.  saying  that  they  depend  on  .  c  source  files  and  also  on  some  file 
called  incl  (which  they  presumably  reference  internally  via  # include). 

To  make  comparison  easier,  we  could  pretend  that  MAKE  used  i.isp  expressions  rather  than  requiring 
a  special  parser.  In  such  ease,  a  makef  1 1  e  might  contain  an  expression  like: 

(DEFINE-FOR-MAKE  MYSYS 
(MYSYS  ("a.o”  "b.o") 

(LOAD- IF -NOT- LOADED  "a") 

(LOAD-IF-NOT-LOADED  "b") 

(LOAD-IF-NOT-LOADED  "m")) 

("a.o"  ("incl"  "a.c") 

(COMPILE-FILE  "a.c")) 

("b.o"  ("incl"  "b.c") 

(COMPILE-FILE  "b.c"))) 

One  problem  with  MAKE  is  that  not  all  the  information  in  a  makefile  is  cxpliciL  For  example, 
the  fact  that  the  system  sources  arc  incl,  a.c,  b.c  is  nowhere  explicit.  There  is  nothing  (other 
than  hoping  that  .  o  files  are  not  source  files  and  anything  else  is)  which  identifies  them  as  die 
source  files.  A  programmer  wanting  to  write  a  utility  for  producing  hardcopy  of  a  system's  sources 
would  not  be  able  to  drive  the  utility  off  of  Hie  information  contained  in  the  makefile. 

MAKE  offers  no  theory  of  how  to  include  or  process  additional  information,  l-br  example,  having 
the  tail  of  each  module  clause  (the  part  after  the  colon  in  the  original  Unix  syntax)  say  how  to 
build  the  module  is  fine  for  a  compiled-only  language,  hut  is  not  reasonable  in  a  language  like 
lisp  which  embraces  die  notion  of  an  interpreter  and  a  compiler  that  can  share  die  load.  There  ia 
no  obvious  way  to  extend  MAKE  in  an  upward-compatible  fashion  in  order  to  allow  specification 
of  commands  for  compiling  modules  as  well  as  commands  for  loading  diem.  There  is  also  no 
provision  for  adding  other  kinds  of  information,  such  as  an  alternate  notion  of  what  it  means  for 
modules  to  be  out  of  date  or  how  to  handle  circular  dependencies. 

In  fact,  MAKE  amounLs  to  little  more  than  a  batch  facility  with  a  simple  hut  inflexible  provision 
for  ignoring  unnecessary  commands.  Just  as  with  normal  hatch  files,  to  do  two  operations  (c.g., 
compile  and  load)  requires  two  batch  files.  While  it  might  be  syntactically  convenient  for  some 
common  applications  in  compiled-only  languages  under  Unix.  MAKE  dues  not  represent  a  theory 
of  how  to  maintain  systems. 


Ihc  DEFSYSTEM  Facility 

The  l.isp  Machine  DEFSYSTEM  facility  (Weinreh  81]  bills  itself  us  a  “general  and  extensible"  tool 
for  maintaining  systems  broken  into  several  files.  It  provides  a  means  of  noting  what  files  belong 
to  what  modules,  what  modules  depend  on  what  other  modules  (both  for  compilation  and  for 
loading),  and  is  extensible  to  allow  the  addition  of  new  system-building  “transformations"  (such  as 
calls  to  alternate  compilers  and  loaders). 

Information  about  the  structure  of  a  system  is  defined  with  the  DEFSYSTEM  special  form.  loiter, 
MAKE-SYSTEM  can  be  called  to  perform  a  pre-defined  set  of  operations  upon  the  system. 

Here  is  a  sample  system  description  written  in  DEFSYSTEM  notation: 

(DEFSYSTEM  MYSYS 

(: MODULE  MACROS  ("incl")) 

(.MODULE  A  ("a")) 

(: MODULE  B  ("b")) 

(: MODULE  ML IB  ("m")) 

(  :  FASLOAD  ML IB) 

(: COMPILE-LOAD  MACROS) 

( :COMPILE-LOAD-INIT  A  (MACROS) 

( : FASLOAD  MACROS) 

(: FASLOAD  ML IB)) 

(:COMPILE-LOAD-INIT  B  (MACROS) 

(: FASLOAD  MACROS) 

(: FASLOAD  ML IB  A))) 

The  : MODULE3  clauses  specify  which  files  belong  to  each  of  the  modules.  For  example,  the  INCL 
module  refers  to  the  file  " incl .  1 i sp",  the  A  module  refers  to  "a.  1  isp",  etc. 

The  :  FASLOAD  clause  for  ML  IB  says  that  "m.  1  isp"  is  a  standard  library  which  must  be  loaded. 
T  his  definition  doesn’t  specify  when  it  might  be  recompiled;  presumably  that  is  handled  by  some 
other  agency. 

The  :COMPILE-LOAD  clause  for  MACROS  says  that  "incl. lisp"  is  part  of  the  system  and  must 
be  recompiled  if  changed,  but  it  has  no  particular  preconditions  to  be  satisfied  prior  to  compilation 
or  loading. 

The  :  COMPILE -LOAD- IN  IT  clauses  for  A  and  B  specify  various  kinds  of  dependency  information. 
Roth  must  be  recompiled  not  only  if  they  themselves  change,  but  also  if  MACROS  changes.  Before 
compiling  A,  one  must  first  load  MACROS:  before  loading  A,  one  must  first  load  ML  IB.  Before 
compiling  B,  one  must  first  load  MACROS;  before  loading  B,  one  must  first  load  ML  IB  and  A. 

Although  names  like  :COMPILE-LOAD  may  look  like  function  names,  they  arc  not.  They 
simply  declare  that  compilation  and/or  loading  may  need  to  occur  under  certain  circumstances. 
DEFSYSTEM  notation,  unlike  that  of  MAKE,  is  declarative  rather  than  procedural.  Because  of 
this,  MAKE-SYSTEM  can  perform  more  than  one  kind  of  operation  (for  each  DEFSYSTEM  form), 
where  MAKE  could  only  perform  one  (for  each  makef  i  le).  For  example,  given  certain  arguments, 
MAKE -SYSTEM  will  compile  a  system.  Given  dilVerent  arguments,  it  will  simply  load  an  already- 
compiled  version  of  the  system. 

On  the  other  hand,  tire  set  of  operations  that  MAKE -SYSTEM  will  perform  is  pre-defined  and  not 
extensible.  This  means  that  if  someone  wanted  to  add  a  new  utility  (r.g..  for  hardcopying  sources) 
there  would  be  no  way  to  do  it  because  there  is  no  user-advertised  mechanism  for  asking  for  a 
list  of  a  system’s  source  files.  The  information  is  present  and  is  used  internally  by  various  system 
utilities,  but  is  not  advertised  to  users  as  part  of  the  standard  abstention. 


"'In  Lisp  Machine  lisp,  symbols  preceded  by  a  colon  arc  "keywords”  interned  in  a  canonical  "keyword 
psickage."  By  special  decree,  keywords  arc  sell-quoting  (Le.,  bound  to  themselves),  lienee.  ’ :  F00  anil  :  F00 
evaluate  to  (lie  same  thing.  :  F00. 


Another  gripe  commonly  heard  about  DEF SYSTEM  is  that  it  was  designed  to  handle  hard  eases, 
but  that  for  simpler  tasks  it  is  generally  cumbersome  and  unpleasant  to  use. 

Still  another  problem,  which  contributes  to  the  overall  feeling  of  clumsiness,  is  that  there  is  no 
transitivity  of  dependency  information  between  OEFSYSTEM  modules.  Hence,  if  OEFSYSTEM  is 
told  that  C.LISP  depends  on  B.LISP  and  that  B.LISP  depends  on  A. LISP,  DEFSYSTEM  will 
not  infer  that  C.LISP  depends  on  A. LISP.  Technically,  it  would  be  incorrect  to  do  otherwise 
because  it  may  be  the  ease  that  the  part  of  B.LISP  which  depends  on  A. LISP  is  not  used  by 
C.LISP.  However,  the  result  of  this  decision  on  the  part  of  the  DEFSYSTEM  designers  is  that 
large  OEFSYSTEM  forms  tend  to  take  on  a  pyramidal  shape  as  later  compile-load  specifications  are 
forced  to  specify  an  enormous  number  of  dependencies  explicitly:4 

(defsystem  KBE 

(.-name  "Knowledge-Based-Editor" ) 

(: short-name  "KBE") 

( : module  LET  "KBE:  LISPM2 ;  LET’  :pacfcag«  KBE) 

(module  LETS  "KBE:  LML IB ;  LETS") 

( : module  PP  "KBE:  LML IB;  GPRINT") 

( : module  BASE  ("BASIC"  "NOISE"  "SETS*  "CONDITIONS"  "EN6LISH" 

"SYSTEM-MAGIC"  "FLAVOR-MAGIC")) 

( : module  MACRO  ("MACRO")) 

( : module  KVARS  "KVARS") 


(: fas  load  PP) 

(: fas  load  LET) 

( : fasload  LETS) 

( :comp11o-load-tn1t  BASE  (PP) 

(: fasload  PP  LET  LETS) 

(:fas1oad  PP)) 

(: comp  tie- load- ini t  MACRO  (PP) 

(: fasload  PP  LET  LETS  BASE) 

(: fasload  BASE)) 

(: compile- load- init  KVARS  (PP  MACRO) 

( : fasload  PP  LET  LETS  BASE  MACRO) 

(: fasload  BASE)) 

( : comp  lie- load- Init  KMAC  (PP  MACRO) 

(: fasload  PP  LET  LETS  BASE  MACRO  KVARS) 

(-.fasload  BASE  KVARS)) 

(: comp  lie- load- In It  IMAC  (PP  MACRO  KMAC) 

( : fasload  PP  LET  LETS  BASE  MACRO  KVARS  KMAC) 

(: fas  load  BASE  KVARS  KMAC)) 

( :comp11e-1oad-1n1t  M80X  (PP  MACRO) 

(: fasload  PP  LET  LETS  BASE  MACRO  KVARS  IMAC) 

(: fasload  BASE  KVARS  IMAC)) 

( : comp  lie- load- Init  BEHAVE  (PP  MACRO  IMAC) 

( : fasload  PP  LET  LETS  BASE  MACRO  KVARS  IMAC) 

(: fasload  BASE  KVARS  IMAC  MUOX)) 

( :compt1e-load-1nit  KUTIL  (PP  MACRO  (MAC  BEHAVE) 

(: fasload  PP  t  ET  IETS  BASE  MACRO  KVARS  IMAC  BEHAVE) 
(: fas  load  BASE  KVARS  IMAC  MUOX  BEHAVE)) 


( :comp11e-1oad-  init  MAGIC  (PP  MACRO  IMAC  BEHAVE  SIMPLE  LD1  SOCIETY) 

( : fasload  PP  HI  1EIS  BASE  MACRO  KVARS  IMAC 

UI'HAVI  KUTIL  S1MPIE  EDI  SOCIEIY  SI  WORD  2MAGIC) 

( : fas  load  BASI  KVARS  IMAC  Bf HAVE  KUTIL  EDI 
SOCIETY  SINORU  BRIDGE  ZMAG1C)) 

..., 

'  ll.i,  example  is  taken  directly  from  a  real  pmgiain  system  developed  by  the  MU'  Programmer's  Apprentice 
group.  The  full  definition  is  much  longer,  describing  die  relations  between  about  VO  dies,  grouping  them  into 
about  50  logical  modules. 


0EFSYS1EM  is  extensible,  but  in  addition  to  its  other  problems,  much  of  the  data  available  to 
extension  writers  is  in  the  form  of  special  variables  which  arc  available  while  its  “transformations” 
arc  happening.  To  illustrate  this,  we  include  an  excerpt  from  the  l  isp  Machine  Manual  section 
describing  state  variables  which  can  be  used  in  writing  DEFSYSTEM  extensions: 

SI :*SYSTEM-BEING-MADE*  Variable 

TTic  internal  data  structure  which  represents  the  system  being  made. 

SI : *MAKE -SYSTEM- FORMS -TO-BE -EVALED-BE FORE*  Variable 

A  list  of  forms  which  arc  evaluated  before  the  transformations  are  performed. 

SI :*MAKE -SYSTEM- FORMS-TO-BE -EVALED-AFTER*  Variable 

A  list  of  forms  which  arc  evaluated  after  the  transformations  arc  performed. 

SI : *MAKE -SYSTEM- FORMS-TO-BE -EVALEO- FINALLY*  Variable 

A  list  of  forms  which  arc  cvalutcd  after  the  body  of  MAKE -SYSTEM  has  completed.  This  differs 
from  SI:*MAKE-SYSTEM-FORMS-TO-BE-EVALED-AFTER*  in  that  these  forms  are  evaluated 
outside  the  “compiler  context,"  which  sometimes  makes  a  difference. 

SI : *QUERY-TYPE*  Variable 

Controls  how  questions  arc  asked.  Its  normal  value  is  :  NORMAL.  :  NOCONFIRM  means  no  questions 
will  be  asked  and  SELECTIVE  asks  a  question  for  each  individual  file. 


(SI : DEF I NE -MAKE -SYSTEM-SPECIAL- VARIABLE  var  val  [ defvar-p ])  Special  Form 
Causes  var  to  be  bound  to  val,  which  is  evaluated  at  MAKE -SYSTEM  time,  during  the  body  of  the 
call  to  MAKE -SYSTEM.  This  allows  you  to  define  new  variables  similar  to  those  listed  above.  If 
defvar-p  is  specified  as  (or  defaulted  to)  T,  var  is  defined  with  DEF  VAR.  It  is  not  given  an  initial 
value.  If  defvar-p  is  specified  as  NIL,  var  belongs  to  some  other  program  and  is  not  DEFVAR’cd 
here. 

Because  information  is  proccdurally  embedded  in  this  way,  it  may  not  be  explicit  at  arbitrary  times. 

Also,  it  imposes  a  lot  of  predefined  mechanism  for  talking  about  systems  which  might  not  be 
convenient  or  even  relevant  in  certain  situations. 

These  problems  with  DEFSYSTEM’s  design  make  it  difficult  to  develop  tools  which  interface  to 
those  provided  with  DEFSYSTEM.  The  net  result  is  that  the  DEFSYSTEM  formalism  is  far  less 
flexible  than  we  might  want  it  to  be. 

IV.  Proposal 

Both  DEFSYSTEM  and  MAKE  offer  interesting  functionality,  but  that  functionality  is  tightly  bonded 
to  the  accompanying  syntax.  Neither  seeks  to  provide  a  theory  of  how  to  allow  specification  of  a 
system  without  binding  the  user  to  a  particular  syntax. 

No  matter  what  the  nature  of  the  specification  language,  llic  kinds  of  high  level  operations  to 
be  performed  upon  systems  is  not  going  to  vary.  We  will  still  want  to  edit,  compile,  load,  and 
hardcopy  systems.  Hence,  rather  than  propose  yet  another  system  description  language  with  a  new. 
improved  notation,  we  will  propose  a  framework  in  which  syntax  and  functionality  are  permitted 
to  vary  independently. 

In  this  section,  we  review  the  details  of  the  proposal,  but  its  essence  is  the  suggestion  that  associated 
with  every  system  there  should  be  some  object  which  responds  appropriately  to  a  pro  defined  set 
of  operations  which  support  appropriate  maintenance  of  the  system.  I’m  another  way.  we  assert 
that  a  key  problem  in  previous  system  maintenance  utilities  is  that  they  cither  were  not  object 
oriented  or  did  not  use  their  object  oriented  nature  to  their  best  advantage. 


Any  sort  of  generic  operations  facility  (<*.&.  that  provided  in  T  (Rees  84|  or  Act- 1 1  ITheriault 
83j)  would  suffice  to  implement  this  proposal.  Wherever  possible,  we  will  use  an  abstract  function* 
calling  syntax  for  presentation.  In  places  which  call  for  examples  from  a  particular  language,  we 
will  use  Lisp  Machine  Lisp  for  presentation  purposes. 

Basic  Protocol 

By  studying  operations  that  are  typically  performed  upon  files,  we  can  make  a  list  of  the 
common  operations  we  might  expect  to  be  performed  upon  systems.  One  obvious  choice  is 
the  update  operation,  which  includes  compilation  or  translation  from  one  language  to  another, 
parser  generation,  etc.  Another  is  instantiation,  which  includes  the  loading  of  files  or  die  execution 
of  some  kinds  of  setup  code.  Other  useful  operations  might  be  editing,  hardcopying,  and  archiving. 

To  support  these  operations,  we  define  the  following  functions: 

(SYSTEM: GENERATE -PLAN  system  operation)  -*  actions 

Given  the  name  of  an  abstract  operation  (such  as  : UPDATE.  :  INSTANTIATE,  or  : HARDCOPY), 
returns  a  list  of  actions  (abstract  plan  steps)  which  will  accomplish  die  operation. 

(SYSTEM: EXECUTE-PLAN  system  actions) 

Hxccutcs  a  list  of  actions  (or  plan  steps),  such  as  those  returned  from  a  SYSTEM:  GENERATE -PLAN 
requesL 

(SYSTEM: EXPLAIN-PLAN  system  actions) 

Types  out  an  explanation  of  what  would  be  the  effect  of  executing  actions. 

(SYSTEM: SOURCE-FILES  system)  -»  files 
Returns  a  list  of  the  source  files  for  the  system. 

To  this  basic  seL  we  will  add  the  following  two  functions.  It  might  be  argued  that  they  arc 
superfluous  in  die  presence  of  die  above  functions,  but  in  practice  having  them  will  greatly 
enhance  die  clarity  of  some  code.  By  adding  diem  to  the  standard  protocol,  we  encourage  a  clearer 
programming  style  and  standardize  on  terms: 

(SYSTEM: EXECUTE-ACTION  system  action) 

Hxccutcs  a  given  action  (plan  step), 

(SYSTEM: EXPLAIN-ACTION  system  action) 

Types  out  an  explanation  of  what  would  be  the  effect  of  executing  action. 

For  convenience,  well  also  define  diat  if  the  argument  given  to  EXPLAIN  or  EXECUTE  is  the  name 
of  an  operation  rudicr  than  a  set  of  steps,  dicn  die  plan  will  be  generated  automatically.  This 
allows  us  to  say: 

( SYSTEM: EXECUTE -PLAN  system  : UPDATE ) 
and  (SYSTEM: EXPLAIN-PLAN  system  : UPDATE ) 

where  formerly  we  would  have  had  to  say: 

(SYSTEM : EXPLAIN-PLAN  system  ( SYSTEM: GENE RATE -PL AN  system  :UPOATE)) 
and  (SYSTEM: EXECUTE-PLAN  system  (SYSTEM: GENE RATE -PLAN  system  :UPDATE)j. 

Defining  systems 

Under  our  proposal,  systems  arc  described  using  the  DEFINE -SYSTEM  special  form.  It  creates  a 
system  object  ami  stores  it  globally  lor  use  at  a  later  lime.  It  has  die  syntax: 

(DEFINE-SYSTF.M  name  type  .  options). 


ITtc  exact  nature  of  the  options  will  vary  depending  on  the  type  of  the  system.  For  some  systems, 
it  may  just  be  a  list  of  files.  For  others,  it  might  be  a  more  complex  data  structure  specifying 
specific  dependency  information.  This  proposal  is  designed  explicitly  to  avoid  taking  a  stand  on 
what  goes  in  this  portion  of  the  system  specification. 

To  support  this  kind  of  type-specific  option  processing,  we  need  functions  to  digest  a  type-specific 
options  list: 

(SYSTEM: PROCESS-OPTIONS  system  options-! 1st) 

Processes  an  options-list.  such  as  die  body  of  a  DEFINE-SYSTEM  form.  This  might,  but  need  not 
necessarily,  be  done  by  mapping  SYSTEM: PROCESS-OPTION  down  die  options-list. 

(SYSTEM: PROCESS-OPTION  system  name  .  data ) 

Processes  a  single  option  with  given  name  and  data.  The  option  name  :  NAME  must  be  handled. 
Handling  of  any  other  option  is  at  die  discretion  of  the  particular  system  type. 

Creating  System  Objects 

The  DEFINE-SYSTEM  special  form  is  supported  by  a  normal  function,  called  CREATE-SYSTEM, 
which  has  die  syntax: 

(CREATE-SYSTEM  name  type  options). 

CREATE-SYSTEM  returns  an  object  representing  the  system,  but  docs  not  store  it  in  any  global  place. 

Such  an  object  is  called  an  anonymous  system.  CREATE-SYSTEM  (and  hence  DEFINE-SYSTEM) 
works  by  creating  an  object  of  die  designated  system  flavor  and  then  calling  appropriate  functions 
to  set  its  name  and  process  its  options. 

Extensibility  ' 

To  be  appropriately  extensible,  each  implementation  would  have  to  define  how  these  functions 
related  to  the  generic  operations  facility  provided  by  that  language.  For  example,  on  die  Lisp 
Machine,  die  interface  to  flavors  might  look  like: 

(DEFUN  SYSTEM: PROCESS-OPTIONS  (SYSTEM  OPTIONS-ALIST) 

(SEND  SYSTEM  : PROCESS-OPTIONS  OPTIONS-ALIST)) 

(DEFUN  SYSTEM: PROCESS-OPTION  (SYSTEM  NAME  &REST  DATA) 

(LEXPR-SEND  SYSTEM  : PROCESS-OPTION  NAME  DATA)) 


F.xploiting  Inheritance 

Languages  which  provide  generic  operations  and  facilities  for  type  inheritance  would  probably  offer 
at  least  two  pre-defined  types. 

A  type  called  SYSTEM  should  be  at  the  base.  It  should  have  no  properties  other  than  identifying 
the  object  as  a  system.  Designers  who  wish  to  start  over  “from  scratch"  in  designing  new  types 
of  systems  adhering  to  the  protocol  we  propose  would  start  with  the  SYSTEM  type  and  work  from 
there. 

Another  type,  which  we  shall  call  VANILLA-SYSTEM,  might  offer  some  very  general  functionality 
which  might  be  of  use  to  many  kinds  of  systems.  Using  whatever  inheritance  mechanism  was 
appropriate  to  the  language,  implementors  of  many  new  system  types  might  be  able  to  inherit  from 
this  slightly  less  general  type  rather  than  starting  from  scratch  and  building  their  system  type  from 
type  SYSTEM,  Naturally,  VANILLA-SYSTEM  would  inherit  from  SYSTEM. 

In  the  Lisp  Machine,  for  example,  customizing  can  frequently  be  done  by  mixing  VAN  ILL  A -SYSTEM 
into  the  new  flavor  and  adding  or  changing  a  few  methods.  Nothing  prevents  the  designer  from 
starting  from  scratch  and  implementing  all  the  methods  from  scratch;  but  this  will  typically  involve 
more  work  than  is  necessary. 


A  typical  system  flavor  might  look  like: 

(DEF FLAVOR  system-type  ( ...  Instance  variables...) 

(VANILLA-SYSTEM)) 

followed  by  definitions  of  new  or  customized  methods. 

For  example,  on  the  Lisp  Machine,  the  : PROCESS-OPTIONS  method  might  be  expected  to  be 
defined  by: 

(DEFMETHOD  (VANILLA-SYSTEM  rPROCESS-OPTIONS)  (OPTIONS) 

(DOLIST  (OPTION  OPTIONS) 

(SEND  SELF  : PROCESS-OPTION  OPTION))) 

This  is  defined  as  part  of  VANILLA-SYSTEM  to  save  everyone  the  trouble  of  writing  that  same 
method.  In  fact,  VANILLA-SYSTEM  might  even  define  :  PROCESS-OPTION  to  use  :CASE  method 
dispatch  (so  that  handling  each  kind  of  option  may  be  defined  by  a  separate  DEFMETHOD  form). 

If  so,  we  might  expect  to  also  find  definitions  such  as  these  in  VANILLA-FLAVOR: 

(DEFMETHOD  (VANILLA-SYSTEM  :CASE  : PROCESS-OPTION  : FULL-NAME)  (NAME) 

(SETQ  FULL-NAME  (STRING  NAME))) 

(DEFMETHOD  (VANILLA-SYSTEM  :CASE  .PROCESS-OPTION  : SHORT-NAME)  (NAME) 
(SETQ  SHORT-NAME  (STRING  NAME))) 

(DEFMETHOD  (VANILLA-SYSTEM  -.CASE  : PROCESS-OPTION  : NAME )  (NAME) 

(IF  (NOT  SHORT -NAME)  (SETQ  SHORT -NAME  (STRING  NAME))) 

(IF  (NOT  FULL-NAME)  (SETQ  FULL-NAME  (STRING  NAME)))) 

i  / 

(DEFMETHOD  (VANILLA-SYSTEM  OTHERWISE  : PROCESS-OPTION)  (NAME  &REST  DATA) 
(FERROR  "Bad  option:  ~S~%Data:  ~S”  NAME  (COPYLIST  DATA))) 

Presumably,  systems  inheriting  from  VANILLA-SYSTEM  would  define  additional  :  PROCESS-OPTION 
methods  for  any  specifications  appropriate  to  them. 

Modular  Extensions 

Given  these  basic  facilities,  it  is  easy  to  make  modular  extensions.  The  function  HARDCOPY-SYSTEM 
discussed  earlier  could  be  written  simply  as: 

(DF.FUN  HARDCOPY-SYSTEM  (SYSTEM) 

(MAPC  #' HARDCOPY-FILE  (SYSTEM:SOURCE-FILES  SYSTEM))) 

Likewise,  a  facility  for  saving  a  snapshot  of  a  system’s  source  files  to  another  directory  might  be 
written: 

(DEFUN  ARCHIVE-SYSTEM  (SYSTEM  ARCHIVE-DIRECTORY) 

(DOLIST  (FILE  (SYSTEM.-SOURCE-FILES  SYSTEM)) 

(COPY-FILE  FILE  ARCH I VE -01 RECTORY ) ) ) 

There's  no  reason  the  user  should  necessarily  have  to  write  things  like  this  himself.  In  general,  it’s 
nice  to  have  system  libraries  that  have  this  sort  of  thing  pre-defined.  The  important  thing  is  that 
if  they  were  not  primitively  provided,  they  would  be  no  problem  to  write  as  extensions  because  a 
basic  set  of  operations  has  been  chosen  which  lends  itself  to  modular  extension. 

Kven  the  fancier  options  to  DEFSYSTEM,  such  as  the  :  SELECTIVE  option,  fall  directly  out  of  this 
modularization.  For  example,  the  essence  of  the  :  SELECTIVE  option  is  captured  by: 

(LCT  ((PLAN  ( SYSTEM: GENE  RATE -PLAN  system  :UPDATE ) ) ) 

(FORMAT  T  "~&To  update  ~A:"  system) 

(DOLIST  (STEP  PLAN) 

(SYSTEM: E XPL AIN- ACT  ION  system  STFP)) 

(UNLESS  (NOT  (Y-OR-N-P  "Ready  to  go  ahead?  ")) 

(SYSTEM: EXECUTE -PLAN  system  PLAN))) 


Variations  arc  also  simple.  For  example,  per-step  querying  could  be  achieved  by: 

(DOLIST  (STEP  (SYSTEM: GENE RATE -PLAN  system  : UPDATE ) ) 

(SYSTEM: EXPLAIN-ACTION  system  STEP) 

(UNLESS  (NOT  (Y-OR-N-P  "OK?  ")) 

(SYSTEM: EXECUTE-ACTION  system  STEP))) 

Systems  with  Simple  Dependencies 

Until  now,  not  much  has  been  said  about  what  kind  of  information  should  go  into  the  options 
portion  of  a  DEFINE -SYSTEM  form. 

In  the  simplest  case,  all  we  might  want  to  specify  is  the  set  of  files  involved.  So,  for  example,  we 
might  imagine  a  kind  of  system  called  SIMPLE-SYSTEM  in  which  the  options  section  was  just  a 
list  of  files,  so  that  die  system  we  earlier  specified  by: 

(DEFINE-SYSTEM-SOURCES  ’MYSYS 

"MACROS. LISP"  "UTILITIES. LISP"  "MAIN. LISP") 

would  now  be  specified  by: 

(DEFINE-SYSTEM  MYSYS  SIMPLE-SYSTEM 

"MACROS. LISP"  "UTILITIES. LISP"  "MAIN . LISP" ) . 

Contrast  the  simplicity  of  this  approach  with  the  MAKE  specification  of: 

(DEFINE-FOR-MAKE  MYSYS 

("main.bin"  ("uti1ities.bin"  "macros.bin"  "main. lisp") 
(LOAD-IF-NOT-LOADED  "macros.bin") 

( LOAD -IF- NOT - LOADED  "utilities.bin") 

(LOAD  (COMPILE-FILE  "main. lisp"))) 

("utilities.bin"  ("macros.bin"  "utilities. lisp") 

(LOAD-IF-NOT-LOADED  "macros.bin") 

(LOAD  (COMPILE-FILE  "utilities. lisp"))) 
("macros.bin"  ("macros. lisp") 

(LOAD  (COMPILE-FILE  "macros . 1 isp" ))) ) 

or  the  OEFSYSTEM  form: 

(DEFSYSTEM  MYSYS 

(.MODULE  MACROS  ("macros")) 

(.MODULE  UTIL  ("utilities")) 

( : MODULE  MAIN  ("main")) 

(: COMPILE-LOAD  MACROS) 

(:COMPILE-LOAD-INIT  UTIL  (MACROS) 

( : FASLOAD  MACROS) 

(:FASLOAD  MACROS)) 

( :COMPILE-LOAD-INIT  MAIN  (MACROS  UTIL) 

(: FASLOAD  MACROS  UTIL) 

(: FASLOAD  MACROS  UTIL))). 

The  system  defined  by  this  DEFINE-SYSTEM  form  is  a  first-class  object  which  can  be  inspected 
and  manipulated  by  the  abstraction  functions  proposed  in  the  last  section. 


Systems  with  Complex  Dependencies 

Consider  now  a  system  with  a  set  of  macros  (in  MACROS  .LISP)  that  expand  into  calls  to  functions 
in  some  utility  package  (in  MACRO-SUPPORT .  LISP).  It  should  be  an  abstraction  violation  for  the 
consumers  of  tire  macro  package  to  have  to  know  what  support  is  necessary  for  the  package  to 
run.  Yet  with  OEFSYSTEM.  the  specification  must  be  written: 

(DEFSYSTEM  MYSYS 

(:M00ULE  MACROS  ("macros")) 

'( : MODULE  MACRO-SUPPORT  ("macro-support")) 

(: MODULE  F00  ("foo")) 

( :C0MPILE-L0A0  MACRO-SUPPORT) 

(: COMPILE-LOAD  MACROS) 

( :COMPILE-LOAD-INIT  FOO  (MACROS) 

( : FASLOAD  MACROS) 

( : FASLOAD  MACRO-SUPPORT)) 

...) 

A  user  interested  in  abstraction  might  object  to  having  to  specify  FOO’s  dependency  upon 
MACRO-SUPPORT  explicitly.  Certainly  users  of  DEFSYSTEM  have  complained  that  this  lack  of 
abstraction  makes  very  large  systems  very  hard  to  specify  and  maintain  using  DEFSYSTEM. 

Armed  with  our  new  DEFINE -SYSTEM  proposal,  a  new  notation  could  be  developed  to  handle 
die  situation.  We  might,  for  example,  propose  a  notation  where  rather  than  say  “FOO  depends  on 
MACRO-SUPPORT”  explicitly  (as  happens  in  DEFSYSTEM),  we  could  say  “Using  MACROS  causes  a 
need  for  MACRO-SUPPORT.”  Such  a  notation  might  look  like: 

(DEFINE-SYSTEM  MYSYS  MODULAR-SYSTEM 

( : MODULE  MACRO-SUPPORT  ("macro-support")) 

(: MODULE  MACROS  ("macros") 

(: CAUSES 
(  NEEDS 

(: INSTANTIATE  MACRO-SUPPORT)))) 

(: MODULE  FOO  ("foo") 

(: NEEDS 

(: UPDATE  MACROS))) 

...) 

To  argue  lor  or  against  some  particular  new  notation  is  not  the  point  of  this  paper.  Iltc  real 
point  is  that  the  proposed  framework  provides  a  means  of  introducing  alternate  notations  in  a  way 
that  docs  not  interfere  with  existing  notations  and  tools.  Kxisting  tools  can  operate  correctly  upon 
systems  created  with  new  notations  such  as  this  because  it  is  the  functional  behavior  of  systems 
which  hits  been  standardized,  not  the  notation. 

In  (Robbins  84|,  still  another  notation  (to  accompany  a  tool  called  BUILD)  is  proposed  for  specifying 
module  dependency  information.  Although  the  data  abstractions  proposed  in  this  paper  were  not 
designed  with  HUILD  in  mind,  they  seem  appropriate  to  support  it  anyway.  Had  the  proposed 
framework  already  been  in  effect,  it  would  probably  have  been  considerably  simpler  for  Robbins 
to  experiment  with  his  new  notation. 


I WCWIW3 


Systems  with  “Idiosyncratic"  Dependencies 

Some  systems  may  have  very  complicated  file  dependencies,  in  some  eases,  for  example,  code  may 
have  evolved  in  a  way  which  demands  that  an  earlier  copy  of  itself  be  loaded  in  order  to  support  its 
compilation  or  instantiation.  The  description  of  such  systems  and  how  they  arc  to  be  constructed 
may  require  a  complex  notation.  This  is  acceptable  only  if  it  docs  not  affect  the  simplicity  of 
notation  used  to  describe  simple  systems. 

Some  users  have  suggested  that  the  complexity  of  DEFSYSTEM  stems  from  the  fact  that  the  same 
noiationa!  devices  must  be  used  for  all  kinds  of  systems,  whether  simple  or  complex.  In  this  new 
proposal,  systems  can  be  classified  into  different  kinds,  each  with  their  own  notation.  We  have 
illustrated  that  simple  systems  might  require  only  the  specification  of  the  files  involved  and  nothing 
else,  while  some  more  complex  systems  might  be  specified  in  terms  of  inter-file  dependencies. 

For  the  completely  general  ease,  however,  special  “one-shot"  notations  can  still  be  developed  to 
handle  the  specific  needs  of  situations  which  arc  not  yet  sufficiently  well-understood  to  be  handled 
by  a  more  standard  notation.  For  example, 

(OEF FLAVOR  HAIRY-SYSTEM  ()  (VANILLA-SYSTEM)) 

(OEFMETHOO  (HAIRY-SYSTEM  :CASE  -.GENERATE-PLAN  -.UPDATE)  () 

'  ( ( : LOAD  "Foo") 

( : COMPILE  "Foo”)  ;Needs  self  to  compile 
( : LOAD  "Foo") 

( : LOAD  "Bar"))) 

(DEFINE-SYSTEM  MY -HAIRY-SYSTEM  HAIRY-SYSTEM)  ;needs  no  options 

(SYSTEM: EXECUTE-PLAN  (SYSTEM  'MY-HAIRY-SYSTEM)  sUPDATE) 

A  slight  generalization  of  this  idea  leads  to  another  kind  of  system,  which  allows  die  system 
maintaincr  to  specify  explicitly  how  to  handle  each  kind  of  : GENERATE-PLAN  request: 

(DEFFLAVOR  PREPLANNED-SYSTEM  ((FILES  '()) 

(PLANS  •())) 

(VANILLA-SYSTEM)) 

(DEFMETHOD  (PREPLANNED-SYSTEM  : PROCESS-OPTIONS)  (OPTIONS) 

(SETQ  FILES  (CAR  OPTIONS)) 

(SETQ  PLANS  (CDR  OPTIONS))) 

(DEFMETHOD  (PREPLANNED-SYSTEM  :GENERATE-PLAN)  (OPERATION) 

(LET  ( ( P  (ASSQ  OPERATION  PLANS))) 

(COND  (P  (CDR  P)) 

(T  (FERROR  "No  plan  for  operation  ~S"  OPERATION))))). 

Having  done  this,  the  same  system  could  be  re-written: 

(DEFINE-SYSTEM  MY-HAIRY-SYSTEM  PREPLANNED-SYSTEM 
("Foo"  "Bar") 

( -.UPDATE 

( : LOAD  "Foo") 

( : COMPILE  "Foo")  ;Needs  self  to  compile 
( : LOAD  "Foo") 

( : LOAD  "Bar")) 

T  his  notation  has  some  of  die  character  of  the  batch  files  discussed  earlier,  hut  is  much  better 
integrated  with  existing  tools.  Users  of  a  system  could  load  or  compile  it  without  knowing  how 
it  was  defined.  I  atcr,  if  system  dependencies  changed  or  if  a  new  notation  became  available,  the 
system’s  specification  could  be  changed  without  notifying  the  users. 


It 


Systems  with  Dependencies  to  be  Inferred  by  the  Compiler 

Some  people  have  objected  to  the  idea  that  a  system  description  language  should  be  needed  at 
all.  Ilieir  claim  is  that  the  compiler  should  somehow  be  able  to  infer  compilation  dependencies 
by  recognizing  and  recording  assumptions  made  during  compilation. 

litis  sort  of  inference  is  not  completely  reliable  because  compilers  cannot  always  accurately  detect 
compilation  dependencies  induced  by  changes  to  the  suite  of  the  global  environment  during 
compilation,  lire  problem  stems  from  the  presence  in  l  isp  of  powerful  snitc-affccting  primitives 
such  as  EVAL-WHEN  and  the  general  ability  of  macros  to  read  or  alter  global  state  during 
compilation. 

To  see  the  problem,  consider  a  data-driven  macro  facility  such  as  the  following,  which  maintains 
its  suite  both  in  the  compiler  and  in  the  runtime  environment: 


C . LISP 

If  the  definition  of  EXPAND-DEFINITION  had  changed,  it  would  obviously  have  to  be  recompiled. 
On  the  other  hand,  the  definition  of  DEFINE,  while  it  directly  refers  to  EXPAND-DEFINITION, 
is  not  affected  by  the  change  to  EXPAND-DEFINITION.  So  A. LISP  and  C.LISP  would  have  to 
be  recompiled,  but'B.LISP  would  not 

To  complicate  matters,  however,  suppose  that  instead  of  the  above  definition,  EXPANO-DEF INIT  ION 
had  been  defined  by: 

(DEFUN  FXPANO-DEFINITION  (OP  NAME  BODY) 

(FUNCALL  (GET  OP  ’EXPANDER)  NAME  BODY)) 

In  that  ease,  a  seemingly  unrelated  change  such  as  a  change  to  a  definition  elsewhere  such  as: 

(DEFUN  ( : PROPERTY  DEFINE  EXPANDER)  (NAME  BODY)  •••) 
could  affect  the  expansion  of  F00,  causing  C .  LISP  to  need  recompilation. 

There  might  be  a  temptation  to  suggest  that  the  compiler  notice  dial  the  second  argument  to  GET 
in  EXPAND-DEFINITION  is  the  constant  symbol  EXPANDER  and  that  a  function  stored  on  an 
fXI'ANDCR  property  was  changed.  Some  special  cases  might  be  handled  this  way,  but  in  general 
the  problem  can  become  arbitrarily  complex  and  a  correct  analysis  may  be  uncomputablc.  Consider 
the  dilliculty  required  in  understanding  the  implications  even  of: 


(EVAL-WHEN  (EVAL  COMPILE) 

(DEFUN  DEFINE-EXPANDER  (NAME  BODY)  •  •) 

(DEFUN  FOO-EXPANDER  ••) 

* 

• 

(MAPC  *' (LAMBDA  (X  Y)  (PUTPROP  X  Y  ’EXPANDER)) 

’(DEFINE  FOO  •••) 

'(DEFINE-EXPANDER  FOO-EXPANDER  ••■)))• 

'Phis  is  the  sort  of  thing  that  we  might  imagine  a  sufficiently  advanced  compiler  being  able  to  do,  but 
we  might  not  be  willing  to  pay  for  the  overhead  needed  to  deduce  the  information.  Performance 
issues  arc  especially  important  in  systems  which  must  perform  interactively  and  allow  for  user 
intervention,  incremental  development,  and  runtime  redefinition.  Ihc  cost  to  the  programmer  of 
having  to  specify  module  dependencies  may  be  very  cheap  in  comparison  with  the  cost  of  his 
having  to  sit  idle  waiting  for  the  machine  to  deduce  them. 

The  detection  of  loadiimc  dependencies  is  made  similarly  difficult  by  the  ability  of  the  programmer 
to  include  unconstrained  toplcvcl  forms  in  a  file,  to  be  executed  at  load  time. 

Of  course,  the  real  problem  is  probably  that  unconstrained  changes  to  global  state  arc  not  well 
understood  and  may  even  be  a  bad  idea.  There  arc  many  active  language  design  efforts  which  seek 
to  show  that  languages  which  employ  global  state  should  just  be  thrown  out  the  window.  Until 
such  efforts  succeed,  however,  the  problem  of  how  to  specify  and  manipulate  large  systems  will 
remain  an  important  one. 

And  if  it  ever  docs  happen  that  languages  become  sufficiently  constrained  that  all  the  dependencies 
can  be  inferred  mechanically,  our  data  abstractions  will  still  provide  a  needed  interface  between  the 
new  dependency-inference  technology  and  die  standard  maintenance  utilities  for  loading,  compiling, 
hardcopying,  etc.  The  only  thing  we  might  expect  to  change  is  that  complicated  notations  for 
describing  systems  might  give  way  to  simpler  ones.  So  even  if  the  dependency  information  were 
complex,  one  might  eventually  only  need  to  write: 

(DEFINE-SYSTEM  MYSYS  ULTIMATE-SYSTEM 

"MACROS. LISP"  "UTILITIES. LISP"  "MAIN. LISP") 

and  the  rest  of  the  information  could  be  inferred  mechanically. 

Such  a  syntax  might  be  sufficiently  attractive  that  people  would  want  to  rewrite  all  their  systems 
using  it.  But  in  die  interim,  while  people  converted  die  definitions  of  dicir  systems  to  use  this 
simpler  notation,  systems  defined  using  older  notations  could  continue  to  work  correctly  and 
compatibly  under  protective  cover  of  our  abstraction  mechanisms. 

In  preceding  examples,  we  have  shown  how  allowing  muldpic  notations  to  coexist  compatibly  in 
the  same  environment  can  be  used  to  allow  system  maintained  flexibility  in  choosing  a  notation 
which  is  appropriate  for  a  particular  application.  Here  we  see  a  second  reason  for  allowing  multiple 
notations:  to  ease  die  transition  from  each  generation  of  system  description  languages  to  die  next. 


V.  Summary 


Wc  have  motivated  die  need  for  system-definition  tools,  specified  some  criteria  which  such  tools 
should  satisfy,  and  proposed  a  set  of  tools  which  satisfy  those  criteria. 

System  maintenance  tools  should  be  data  driven,  allowing  new  tools  to  be  written  as  extensions  to 
the  existing  tools,  driving  off  the  same  data. 

Ilie  tools  should  be  general  purpose,  allowing  arbitrary  kinds  of  systems  to  be  built  from  them. 
However,  the  need  for  generality  should  not  infect  the  notation,  making  common  eases  notationally 
too  complex  to  specify  conveniently. 

Wc  have  suggested  that  these  ends  should  be  achieved  through  a  protocol-based  approach.  The 
proposed  approach  deemphasizes  the  particular  syntax  used  to  specify  a  system  and  emphasizes 
die  importance  of  making  systems  with  a  well-defined  functional  behavior. 

The  proposal  provides  for  die  construction  of  systems  which  sadsfy  a  pre-deflned  functional 
protocol  This  protocol  allows  system  maintenance  utilities  to  access  and  manipulate  the  system 
specification.  The  proposed  functions  provide  an  interface  for  finding  what  files  make  up  a  system 
and  inquiring  about  how  to  perform  system  maintenance  operations  such  as  editing,  compilation, 
instantiation,  and  liardcopying. 

Because  a  system  can  be  asked  to  produce  a  plan  for  an  operation  such  as  compilation  without 
actually  performing  the  operation,  it  is  possible  to  write  programs  which  inspect  the  plan,  possibly 
optimizing  it  or  presenting  it  for  interactive  approval,  before  executing  it 

The  proposal  also  provides  for  the  possibility  of  having  multiple  system  description  languages 
available  in  the  same  environment  at  the  same  time,  This  capability  allows  a  system  maintainor 
die  frecdoom  to  choose  the  notation  which  is  right  for  a  given  application,  widiout  requiring  those 
who  need  to  manipulate  (compile,  load,  etc.)  the  system  to  know  which  notation  was  used. 

Examples  have  been  given  to  illustrate  how  die  various  features  of  this  proposal  work  together  in 
a  variety  of  situations  to  provide  usefulness  and  flexibility. 


References 

[Bonanni  77)  L.  E.  Ronanni  and  A.  L  Glasser.  "SCCS/PWB  User's  Manual,"  Bell  laboratories, 
Murray  Hill,  NJ,  November,  1977. 

(Feldman  78]  S.  I.  Feldman,  “Make— A  Program  for  Maintaining  Computer  Programs,"  Bell 
laboratories.  Murray  Hill,  NJ.  August,  1978. 

(Miickinlay  84]  J.  Mackinlay  and  M.  Gcncscrcth.  “Kxpressivcncss  of  languages,"  Proceedings  of 
the  National  Conference  on  Artificial  Intelligence .  University  of  Texas,  Austin.  TX.  August,  1984. 

(Rees  84]  J.  Rees.  N.  Adams  and  J.  Meehan,  The  T  Manual Computer  Science  Department,  Yale 
University,  New  Haven,  Cl’.  January,  1984. 

(Robbins  84]  R.  Robbins,  BUILD— A  System  Construction  Tool,  Working  Paper  261.  Artificial 
Intelligence  Laboratory,  MIT,  Cambridge,  MA,  1984. 

[Steele  84]  G.  L.  Steele,  Jr.,  Common  LISP:  The  language.  Digital  Press,  Burlington,  MA,  1984. 

[Theriault  83)  D.  G.  Theriault.  Issues  in  the  Design  and  Implementation  of  Act2,  Technical  Report 
728,  Artificial  Intelligence  Laboratory,  MIT,  Cambridge,  MA,  June,  1983. 

[Weinreb  81]  D.  Weinreb  and  D.  Moon,  Lisp  Machine  Manual,  Fourth  Edition,  MIT  Artificial 
Intelligence  Laboratory,  July.  1981. 


Acknowledgements 

Dan  Brotsky,  Dan  Carnese,  Henry  Licbcrman,  Chuck  Rich,  Patrick  Sobalvarro,  Dick  Waters,  and 
Dan  Weld  read  drafts  of  this  paper,  providing  support  and  commentary.  Comments  by  Waters 
and  Brotsky,  who  read  multiple  drafts,  played  an  especially  important  role  in  improving  the  clarity 
and  organization  of  my  presentation. 

Stephen  Gildca  provided  useful  background  documentation  and  answered  questions  about  Unix 
and  its  MAKE  facility. 


AppemUx  A:  Example* 


I'hcsc  examples  arc  provided  to  illustrate  how  the  code  in  Appendix  B  might  be  used  in  practice. 


Kxample  1 

Consider  a  system  made  of  three  files,  "foo",  "bar",  and  "baa".  Such  a  system  could  be 
described  conveniently  by  a  QEFINC-SYSTEM  form  using  the  SIMPLE-SYSTEM  notation  as 
follows: 

(def ine-system  Examplel  simple- system 

"oz:p$:<zippy>foo.1 isp"  "bar. lisp”  "baa. lisp") 

It  could  also  be  described  using  the  MODULAR-SYSTEM  notation.  For  example, 

(def Ine-systam  Examplel  modular-system 
(:modu1e  ml  "Ok :ps:<ztppy>foo. 1 isp") 

(: module  m2  "bar. lisp" 

(: needs  (‘.instantiate  ml) 

(: update  ml))) 

(:module  m3  "baz.lisp” 

(meeds  (:  Instantiate  m2) 

(-.update  m2)))) 

In  this  case,  the  SIMPLE-SYSTEM  notation  is  obviously  to  be  preferred  since  it  makes  clear  the 
simplicity  of  the  relationships  between  the  files.  In  other  cases,  however,  SIMPLE-SYSTEM  is  not 
going  to  be  powerful  enough  to  express  the  intermodule  relationships,  as  our  next  example  will 
illustrate. 


Kxample  2 

There  Is  a  file  called  "MACROS"  which  requires  compile  time  support  from  "MACRO-HELPERS*. 
The  result  of  the  expansions  of  these  macros  needs  support  from  "MACRO-SUPPORT- 1"  and 
"MACRO-SUPPORT-2"  at  compile  time  and  "BASIC"  at  runtime.  Users  of  "MACROS"  should 
not  have  to  know  about  these  support  files,  so  wc  want  the  fact  that  any  time  a  module  says  it 
needs  "MACROS",  all  the  other  dependencies  arc  added  implicitly. 

TTicrc  is  a  file  called  "META-MACROS".  The  macros  in  that  file  will  expand  into  calls  to  macros 
in  '  MACROS",  though  its  consumer  should  not  have  to  know  this. 

There  is  a  file  called  "UTILITIES"  which  needs  "MACROS"  at  compile  time  and  "BASE"  at 
runtime,  it  uses  the  functions  in  "BASE"  explicitly,  so  must  specify  an  explicit  dependency  upon 
it  even  though  it  happens  that  "MACROS"  provides  an  implicit  dependency. 

There  isa  file  called  "MAIM"  which  depends  upon  "UTILITIES"  at  runtime  and  "META-MACROS" 
at  compile  time. 


(def ine-system  Example2  modular-system 
(:fu11-name  "The  Second  Example”) 

( : module  base  "OZ:PS:<FOO>BASIC.LISP") 

(: module  macro-helpers  "OZ:PS:<FOO>MACRO-HELPERS.LISP") 

(.‘module  macros  "OZ:PS:<FOO>MACROS.LISP" 

( :needs 

(instantiate  macro-helpers)) 

( .‘causes 
( :  needs 

(.‘instantiate  base) 

(:update  macro-support)))) 

(:modu1e  macro-support  ("0Z:PS:<F00>MACR0-SUPP0RT-1.LISP" 

"OZ : PS : <  FOO>MACRO-SUPPORT-2 . LISP" ) ) 

( : module  meta-macros  "OZ:PS:<FOO>META-MACROS.LISP" 

(: causes 
( : needs 

(: update  macros)))) 

( : module  util  "OZ: PS: <FOO>UTILITIES. LISP" 

( .‘needs 

(: update  macros) 

(instantiate  base))) 

( : module  main  "OZ : PS: <F00>MAIN.LISP” 

( : needs 

( :  instantiate  util) 

(: update  meta-macros)))) 

The  equivalent  in  DEFSYSTEM  would  be: 

(def system  Example2 

(: short-name  "EXAMPLE2" ) 

(: full -name  "The  Second  Example") 

(•.module  BASE  ("OZ:PS:<FOO>BASIC.LISP")) 

( : module  MACRO-HELPERS  ( "OZ : PS : <FOO>MACRO-HELPERS. LISP" ) ) 

( : module  MACROS  ("OZ:PS: <FOO>MACROS.LISP")) 

( : module  MACRO-SUPPORT  ("OZ:PS:<FOO>MACRO-SUPPORT-I.LISP" 

"OZ : PS : <  FOO>MACRO-SUPPORT-2 . LISP" ) ) 

( : module  META-MACROS  ("OZ:PS:<FOO>META-MACROS.LISP")) 

( : module  UTILITIES  ("OZ:PS:<FOO>UTILITIES.LISP")) 

(: module  MAIN  ("OZ:PS:<FOO>MAIN.LISP")) 

(: compile- load  BASE) 

( : compile-load  MACRO-HELPERS) 

( : compile- load  MACROS  NIL  ( : FASLOAD  MACRO-HELPERS)) 

(: compile- load  MACRO-SUPPORT) 

( :compile-load  META-MACROS) 

( :compile-load-init  UTIL  (MACRO-SUPPORT  MACROS) 

(: FASLOAD  MACRO-SUPPORT  MACROS) 

( : FASLOAD  BASE)) 

( : comp il e-load- ini t  MAIN  (MACRO-SUPPORT  MACROS  META-MACROS) 

(: FASLOAD  MACRO-SUPPORT  MACROS  META-MACROS) 
(: FASLOAD  UTIL  BASE))) 

Note,  however,  that  even  in  this  small  example,  wc  can  sec  the  characteristic  pyramidal  shape  that 
DEFSYTEM’s  dependency  clauses  tend  to  lake  on. 


Appem/ix  B:  Code 

Mode: LISP;  Package: USER;  Base: 10;  Ponts :MEDFNB 
; ; : ;  System 
;;;  SYSTEM 

Any  flavor  which  claims  to  satisfy  the  SYSTEM*  protocol  should 
Include  this  flavor  In  Its  component  flavors  list. 

(def flavor  system  ()'■() 

( : requ 1  red-methods 
:process-opt1ons 
: process-option 
: source-files 
:generate-p1an 
:expla1n-plan 
:execute-plan 
:expla1n-act1on 
:execute-act1on)> 


O 


BaaMMfiKasa^^ 


; ; ; ;  Vanilla  System 
;;;  VANILLA-SYSTEM 

;;;  A  vanilla  system  knows  about  names  and  how  to  process  options, 

;;;  but  has  no  Interesting  options  It  Is  willing  to  process  that  would 
;;;  make  It  useful  as  something  to  Instantiate. 

(defflavor  vanilla-system  ((short-name  nil)  (full-name  nil))  (system) 

( :method-comb1 nation  (:case  :base-flavor-last 

:generate-plan 

'.process-option 

:expla1n-act1on 

:execute-act1on)) 

: settable-instance- variables) 

;;;  :NAME  (to  VANILLA-SYSTEM) 

; ; ;  Returns  the  name  of  the  system 

;;;  Long  name  Is  preferred  over  short  name  where  both  are  available, 
(defmethod  (vanilla-system  :name)  ()  (or  full-name  short-name)) 

;;;  : PRINT-SELF  ...  (to  VANILLA-SYSTEM) 

For  debugging  convenience, 

;;;  (PRIN1  mysys)  types  something  like:  #<SYSTEM  "My  System”  343324> 

(PRINC  mysys)  types  something  like:  My  System 

(defmethod  (vanilla-system  :pr1nt-self)  (stream  level  print?  &rest  Ignore) 
level  : Ignored 

(let  ((my-name  (send  self  :name))) 

(cond  ((not  prlnl?)  (format  stream  "~A"  my-name)) 

(t 

(format  stream  "#<~S  -A  ~0>" 

(typep  self)  my-name  (Xpolnter  self)))))) 

;;;  : DESCRIBE  (to  VANILLA-SYSTEM) 

#  •  • 

*  »  » 

Does  that  part  of  the  explanation  relevant  to  the  flavor. 

;;;  Other  flavors  mixing  this  In  should  use  :AFTER  or  :BEFORE 
;;;  daemons  to  modify  this  method. 

(defmethod  (vanilla-system  idescrlhe)  () 

(format  t  "-2&-A  Is  a  system  of  type  -S.-X"  self  (typep  self)) 
self) 


; ; ; ;  Options  Facility 

;;;  : PROCESS-OPTIONS  options  (to  VANILLA-SYSTEM) 

•  *  • 

;;;  Maps  across  the  given  options,  digesting  then. 

(defmethod  (vanilla-system  ‘.process-options)  (options) 

(dollst  (data  options) 

(lexpr-send  self  :process-opt1on  (car  data)  (cdr  data))) 
self) 

;;;  : PROCESS-OPTION  opt-name  .  opt-args  (to  VANILLA-SYSTEM) 

;;;  : NAME  Sets  defaults  for  all  name  types. 

;;;  : SHORT -NAME  Sets  the  short  name  (overrides  :NAME  If  given). 

;;;  : LONG -NAME  Sets  the  long  name  (overrides  :NAME  If  given). 

;;;  otherwise  Signals  an  error. 

(defmethod  (vanilla-system  .-case  :process-opt1on  :name)  (data) 

(If  (not  short-name)  (setq  short-name  (string  data))) 

(If  (not  full-name)  (setq  full-name  (string  data)))) 

(defmethod  (vanilla-system  :case  :process-opt1on  :short-name)  (data) 
(setq  short-name  (string  data))) 

(defmethod  (vanilla-system  :case  :procass-opt1on  :full-name)  (data) 
(setq  full-name  (string  data))) 

(defmethod  (vanilla-system  :otherw1se  ’.process-option)  (key  ftrest  data) 
(ferror  "The  option  ~S  is  not  known  to  >S.~%Oata:  ~S" 
key  self  (copyllst  data))) 


Planning/Executing  Actions 
;;;  :GENERATE-PLAN  action  (to  VANILLA-SYSTEM) 

;;;  This  method  returns  abstract  information  about  how  to  perform 
;;;  a  specified  ACTION.  The  reply  Is  in  the  form  of  a  list  of  the 
;;;  form  ((MSG1  .  MSG-ARGS1)  (MSG2  .  MSG-ARGS2)  ...).  such  that 
;;;  sending  each  MSG  (with  the  given  MSG-ARGS)  to  the  object  in 
;;;  order  will  accomplish  the  action  in  question. 

;;;  : UPDATE  How  to  compile  (or  otherwise  update)  the  system. 

: INSTANTIATE  How  to  load  (or  otherwise  instantiate)  the  system. 

;;;  otherwise  An  error  results  If  the  action  isn't  defined. 

(defmethod  (vanilla-system  :otherwise  :generate-plan)  (key  &rest  data) 
(ferror  "The  object  ~S  does  not  know  how  to  ~S.~5CData:  ~S" 
self  key  (copyllst  data))) 

: EXECUTE-PLAN  plan  (to  VANILLA-SYSTEM) 

•  t  I 

;;;  The  steps  of  the  PLAN  are  executed. 

•  •  • 
tee 

;;;  PLAN  may  be  either  a  plan  name  (a  symbol)  or  a  list  of  step* 

;;;  such  as  that  returned  by  a  : GENERATE-PLAN  message. 

(defmethod  (vanilla-system  :execute-plan)  (plan) 

(cond  ( (symbol p  plan) 

(send  self  :execute-plan  (send  self  :generate-p1an  plan))) 

(t 

(dolist  (step  plan) 

(lexpr-send  self  :execute-action  step)))) 

self) 

; ; ;  : EXPLAIN-PLAN  plan  (to  VANILLA-SYSTEM) 

•  !  • 

The  steps  of  the  PLAN  are  explained. 

•  •  • 

•  •  • 

;;;  PLAN  may  be  either  a  plan  name  (a  symbol)  or  a  list  of  steps 
;;;  such  as  that  returned  by  a  : GENERATE-PLAN  message. 

(defmethod  (vanilla-system  :expla1n-plan)  (plan) 

(cond  ((symbolp  plan) 

(send  self  lexplaln-plan  (send  self  :generate-plan  plan))) 

(t 

(dolist  (step  plan) 

(lexpr-send  self  :exp1a1n-act1on  step)))) 


self) 


;;;  : EXECUTE-ACTION  name  .  args  (to  VANILLA-SYSTEM) 

;;;  : EXPLAIN-ACTION  name  .  args  (to  VANILLA-SYSTEM) 

;;;  Sending  :EXECUTE-ACTION  causes  a  given  action  to  occur. 

Sending  : EXPLAIN-ACTION  describes  what  a  given  action  would 
; ; ;  do  if  performed. 

;;;  These  messages  are  handled  by  :CASE  method  dispatch. 

: LOAD  Load  a  (Lisp)  file. 

;;;  .COMPILE  Compile  a  (Lisp)  file. 

;;;  Otherwise  an  error. 

(defmethod  (vanilla-system  :case  :execute-act1on  : load )  (file) 

(load  file)) 

(defmethod  (vanilla-system  :case  :execute-action  : compile)  (file) 

(compiler :compile-file  file)) 

(defmethod  (vanilla-system  otherwise  -.execute-action)  (key  &rest  data) 
(ferror  "The  action  ~S  is  not  known  to  ~S.~%Data:  ~S" 
key  self  (copylist  data))) 

;;;  : EXPLAIN-ACTION  name  .  args  (to  VANILLA-SYSTEM) 

»  «  « 

;;;  If  the  action  Is  valid  but  a  description  wasn’t  available,  try 
;;;  to  conjure  up  a  plausible  description  based  on  the  name  of  the 
;;;  action  and  its  arguments. 

(defmethod  (vanilla-system  :otherw1se  :explain-act1on)  (key  ftrest  data) 
(cond  ((not  (memq  key  (send  self  :execute-act1on  :wh1ch-operat1ons)>) 

(ferror  "The  action  ~S  Is  not  known  to  ~S,  so  can’t  describe  1t.~ 
~%Data :  ~S" 

key  self  (copylist  data))) 

(t  (format  t  "~&~A~0[  ~{~A~t, 

(string-capital ize-words  key)  data)))) 
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; ; ; ;  Simple  System 
;;:  SIMPLE-SYSTEM 

see 

;:;  A  simple  system  Is  a  system  which  has  left-to-rlght  file 
;;;  dependencies. 

(defflavor  simple-system  ((source-files  •()))  (vanilla-system) 

:gettab1e- Instance- variables) 

;;;  :AFTER  : DESCRIBE  (to  SIMPLE-SYSTEM) 

•  »  • 

;;;  Tacks  on  some  Information  about  the  files  which  make  up  this  system. 

(defmethod  (simple-system  :after  .-describe)  () 

(format  t  "~&It  has  source  f 11es-{-<~X-l: ;  ~S~>~t , -} . -X"  source-files)) 

;;;  : PROCESS-OPTIONS  (to  SIMPLE-SYSTEM) 

•  •  l 

;;;  The  only  options  allowed  to  a  simple  system  Is  a  list  of  file 
;;;  names  with  left-to-rlght  ordering  dependencies. 

(defmethod  (simple-system  :process-opt1ons)  (options) 

(when  options 

(setq  source-files  '()) 

(let  ((default-pathname  (fs:merge-pathname-defaults  (car  options))}) 
(dollst  (file  options) 

(setq  default-pathname 

(fs :merge-pathname-defau1ts  file  default-pathname)) 

(push  default-pathname  source-files)) 

(setq  source-files  (nreverse  source-files))) 
t)) 

; : ;  : GENERATE -PLAN  : UPDATE  (to  SIMPLE-SYSTEM) 

•  •  • 

I  I  I. 

;;;  To  update  this  kind  of  system,  one  must  compile  and  load  each 
;;;  of  Its  files  In  sequence. 

(defmethod  (simple-system  :case  :generate-plan  :update)  () 

(mapcan  it' (lambda  (file) 

(list  (list  ':comp11e  file) 

(list  ':load  (send  file  :new-pathname 

:type  .-bln)))) 

source-files)) 

;;;  : GENERATE-PLAN  : INSTANTIATE  (to  SIMPLE-SYSTEM) 

•  •  • 
tee 

;;;  To  Instantiate  this  kind  of  system,  one  must  simply  load  each 
; ; ;  of  Its  files  In  sequence. 

(defmethod  (simple-system  :case  :generate-p1an  :  Instantiate)  () 

(mapcan  #'( lambda  (file) 

(list  (list  ' : load 

(send  file  :new-pathname  :type  :b1n)))) 

source-files)) 
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;;;;  Pre-Planned  System 
; ; ;  PREPLANNEO-SYSTEM 

;;;  A  preplanned  system  is  a  system  which  has  its  plans  for  manipulation 
specified  explicitly  rather  than  Inferred. 

(defflavor  preplanned-system  ((files  '()) 

(plans  •())) 

(vanilla-system)) 

;;;  -.PROCESS-OPTIONS  (to  PREPLANNED-SYSTEM) 

;;;  The  clauses  in  the  DEFINE-SYSTEM  for  this  kind  of  system  are 
;;;  just  (<p1an-name>  .  <commands>). 

(defmethod  (preplanned-system  :process-options)  (options) 

(setq  files  (car  options)) 

(setq  plans  (cdr  options))) 

;;;  : GENERATE -PLAN  (to  PREPLANNEO-SYSTEN) 

;;;  This  does  simple  table-lookup  to  find  the  plan. 

(defmethod  (preplanned-system  :generate-plan)  (operation) 

(let  ((p  (assq  operation  plans))) 

(cond  (p  (cdr  p)) 

(t  (Terror  "No  plan  for  operation  ~SM  operation))))) 


; ; ; ;  Modular  System 
;;;  MODULAR-SYSTEM 

*  *  I 

;;;  A  modular  system  Is  a  system  which  allows  specification  of 
;;;  Inter-module  dependencies,  both  Implicit  and  explicit. 

(defflavor  modular-system  ((modules  nil))  (vanilla-system) 

: ini  table- Instance- variables 
:gettab1e- Instance- variables 
: settable- Instance-variables) 

;;;  : AFTER  : DESCRIBE  (to  MOOULAR-SYSTEM) 

*  «  i 

;;;  When  a  modular  system  Is  described,  we  tack  on  Information 
;;;  saying  how  many  modules  It  has  and  then  we  ask  each  module 
: ; ;  to  describe  Itself. 

(defmethod  (modular-system  :after  :descr1be)  () 

(let  ((m-11st  (send  self  :modu1e$))) 

(format  t  "-Bit  has  ~D  module-:?:"  (length  m-11st)) 

(dollst  (m  m-llst) 

(send  m  '.describe)))) 

;;;  : GET-MODULE  name  (to  MOOULAR-SYSTEM) 

%  %  • 

*  *  * 

;;;  Returns  the  component  module  with  the  given  name  (or  NIL  If  none). 

(defmethod  (modular-system  :get-modu1e)  (name) 

(dollst  (m  modules) 

(If  (eq  (send  m  -.name)  name)  (return  m)))) 

; ; ;  : SOURCE-FILES 
*  •  • 

Returns  a  list  of  the  source  files  for  the  system. 

(defmethod  (modular-system  :source-f lies)  () 

(apply  ^’append 

(mapcar  #'( lambda  (module) 

(send  module  :source-f lies)) 
modules))) 

;;;  : PROCESS-OPTION  : MODULE  .  spec  (to  MODULAR-SYSTEM) 

♦  *  * 

;;;  Declares  how  to  handle  the  :M0DULE  option.  Creates  an  object 
; : ;  of  type  MODULE  and  lets  It  process  the  associated  spec. 

(defmethod  (modular-system  :case  :process-opt1on  :modu1e)  (Brest  spec) 
(setq  modules 

(nconc  modules 

(neons  (make-instance  'module 

:system  self 

:spec  (copyllst  spec)))))) 


;  Module 
MODULE 


A  module  Is  a  collection  of  files  to  be  used  ee  e  building 
;;;  block  for  modular  systems. 

(defflavor  module  ((name  nil) 

(system  nil) 

(spec  nil) 

(source-files  *()) 

(assertions  *()) 

(needs  *()) 

(causes  *())) 

0 

: Inltable-lnstance-varlables 
:  get table- Instance- variables 
: settable- Instance- variables 

( :method-comb1nat1on  (:case  :base-f lavor-last  :process-assert1on))) 
;;;  : AFTER  :INIT  (to  MODULE) 

See  to  It  that  If  SPEC  was  given.  It  gets  appropriately  proceeded. 

(defmethod  (module  :after  : In  It)  (trest  Ignore) 

(send  self  :process-spec  spec)) 

;;;  : PRINT-SELF  ...  (to  MODULE) 

•  •  • 

•  as 

;;;  For  debugging  convenience. 

(PRIN1  mod)  types  something  like:  #<Modu1e  MYSYS*M0D1  234607> 

;;;  (PRINC  mod)  types  something  like:  MOOi 

(defmethod  (module  :prlnt-self)  (stream  level  print?  Brest  Ignore) 
level  ‘.ignored 

(let  ((my-name  (send  self  :name))) 

(cond  ((not  pr1nl7)  (format  stream  "~A"  my-name)) 

(t 

(format  stream  "#<~S  ~g[~A*-]~*.[Anonymous~‘,~:*~A~]  ~0>" 
(typep  self) 

(let  ((sys  (send  self  :system))) 

(If  sys  (send  Sys  : Short-name))) 
my-name 

(Xpolnter  self)))))) 

;;;  -.DESCRIBE  (to  MOOULE) 

:::  Details  the.  source  files  and  dependency  Information 
; ; ;  for  the  module. 

(deTmethod  (module  :descr1be)  () 

(format  t  "-2&  ~A~d[  ~{~X~A~t, self  source-files) 

(do  ((n  needs  (eddr  n))) 

((null  n)) 

(format  t  "~A  -S  dependenc-BP:  ~(~S~t, 

(car  n)  (length  (cadr  n))  (cadr  n ) ) ) 

(format  t  "-*") 
self) 


: PROCESS-SPEC  spec  (to  MODULE) 

•  •  • 

•  •  • 

Process  the  given  SPEC  absorbing  relevant  Info. 

•  !  5 

;;;  The  NAME  Is  only  absorbed  If  name  Info  Isn't  already  set  up. 

This  Is  because  :PROC£SS-SPEC  may  be  recursively  called  on  others' 

;;;  assertion  lists  If  there  are  Included  modules  with  specs  of  their 
own.  In  such  case,  we  want  to  accept  their  attributes,  but  not 

;;;  their  names. 

•  *  • 
e  e  e 

;;;  The  ASSERTIONS  are  processed  next,  because  presumably  they  specify 
;;;  prerequisites  for  this  module  and  any  files  they  need  loaded  should 
get  set  up  before  we  set  up  the  files  particular  to  this  module. 

*  i  * 

;;;  Finally,  the  FILES  associated  with  this  module  are  processed. 

(defmethod  (module  :process-spec)  (s) 

(when  s 

(If  (not  name)  (setq  name  (car  s))) 

(send  self  :process-assert1ons  (cddr  s)) 

(send  self  :process-f lies  (cadr  s ) ) ) ) 

: PROCESS-FILES  files-list  (to  MODULE) 

•  «  • 

Adds  file  Info  given  In  FILES-LIST  to  the  module's  master  FILES  list. 

(defmethod  (module  :process-f11es)  (file-list) 

(If  (atom  file-list)  (setq  file-list  (list  file-list))) 

(dollst  (file  file-list) 

(cond  ((typep  file  'fs:pathname) 

(setq  source-files  (nconc  source-files  (neons  file)))) 
((strlngp  file) 

(setq  source-files  (nconc  source-files 

(neons  (fs: parse-pathname  file))))) 

((symbolp  file) 

(send  self  :process-spec 

(send  (send  system  :get-module  file)  :spec))) 

(t 

(ferror  "Bad  object  In  file  list:  -S  -  ~S"  file  self))))) 

;;;  : PROCESS-ASSERTIONS  spec  (to  MODULE) 

•  •  • 

•  ft 

;;;  Iterates  across  assertions,  processing  each. 

(defmethod  (module  ’.process-assertions)  (assertion-list) 

(dollst  (assertion  assertion-list) 

(lexpr-send  self  :process-assert1on  assertion))) 
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;:;  : PROCESS-ASSERTION  .  data  (to  NODULE) 

♦  •  • 

•  •  • 

;;;  This  method  Is  used  to  process  dependency  assertions,  etc. 

;;;  Tor  the  given  module.  It  uses  case  method  dispatch: 

•  it 

: NEEDS  Declares  need  to  Instantiate  modules  at  certain  times. 

: CAUSES  Declares  assertions  to  be  forwarded  to  the  consumer. 

;;;  Otherwise  an  error. 

(defmethod  (module  :case  :process-assert1on  .-needs)  (ftrest  data) 

(let  ((pi  (locf  needs))) 

(dollst  (Item  data) 

(let  ((marker  (car  Item))) 

(dollst  (module-name  (cdr  Item)) 

(when  (not  (memq  module-name  (get  pi  marker))) 

(let  ((m  (send  (send  self  :system)  : get-module  module-name))) 
This  may  be  overly  conservative,  but  will  work... 

(send  self  :process-assert1ons  (send  m  :causes)) 

(setf  (gat  pi  marker) 

(nconc  (get  pi  marker)  (neons  module-name))))}))))) 

(defmethod  (module  :case  :process-assert1on  :causes)  ((rest  data) 

::  Filtering  this  Is  technically  unnecessary,  but  It  will  keep 
;;  redefinition  from  swamping  us. 

(dollst  (Item  data) 

(If  (not  (mem  J'equal  Item  causes)) 

(setf  causes  (nconc  causes  (neons  Item)))))) 

(defmethod  (module  :otherw1se  :process-assert1on)  (key  ftrest  data) 

(ferror  "The  ~S  assertion  Is  not  known  to  ~S.~XData:  ~S” 
key  self  (copyllst  data))) 


: ; ; ;  User  Interface 

;;;  (CREATE-SYSTEM  name  type  [options]) 

;;;  Creates  a  system  object  of  the  given  TYPE,  Initializing  It  with 

;;;  with  the  given  NAME  and  OPTIONS.  Returns  the  created  object  without 

;;;  storing  It  permanently  anywhere. 

(defun  create-system  (name  type  Soptlonal  options) 

(let  ((system  (make-instance  type))) 

(send  system  :process-opt1on  :name  name) 

(send  system  :process-opt1ons  options) 
system)) 

;;;  (SYSTEM  name) 

;;;  Gets  the  definition  of  some  globally  defined  system  object. 

(defsubst  system  (name)  (get  name  ’system)) 

(DEFINE-SYSTEM  name  type  .  options) 

;;;  Creates  and  Initializes  a  system  with  the  given  name. 

Stores  the  definition  globally  for  access  later. 

(defmacro  def Ine-system  (name  type  &body  data) 

'(setf  (system  ’.name) 

(create-system  '.name  '.type  ’.data))) 


; ; ; ;  Utility  functions 

(defun  process-options  (system  optlons-allst) 

"Tells  SYSTEM  to  process  the  given  OPTIONS-ALIST." 

(send  system  :process-opt1ons  optlons-el 1st)) 

(defun  process-option  (system  option-name  Arest  option-data) 

"Tells  SYSTEM  to  process  an  Individual  option,  given  Its  NAME  and  DATA." 
(lexpr-send  system  :process-opt1on  option-name  option-data)) 

(defun  source-files  (system) 

"Returns  a  list  of  the  source  files  for  SYSTEM." 

(declare  (values  files)) 

(send  system  :source-f11es)) 

(defun  plan  (system  operation) 

"Returns  a  list  of  ACTIONS  (plan  steps)  for  doing  OPERATION." 

(declare  (values  actions)) 

(send  system  :generate-plan  operation)) 

(defun  execute  (system  actions) 

"Tells  SYSTEM  to  execute  the  given  ACTIONS  (plan  ateps)." 

(send  system  :execute-p1an  actions)) 

(defun  explain  (system  actions) 

"Tells  SYSTEM  to  explain  the  given  ACTIONS  (plan  steps)." 

(send  system  :expla1n-p1an  actions)) 

(defun  execute-action  (system  action) 

"Tells  SYSTEM  to  execute  the  given  ACTION  (plan  step)." 

(send  system  :execute-act1on  action)) 

(defun  explaln-actlon  (system  action) 

"Tells  SYSTEM  to  explain  the  given  ACTION  (plan  step)." 

(send  system  :expla1n-act1on  action)) 


